{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "!!! note\n",
    "\n",
    "    This notebook is intended for usage alongside the second [Bias in Data Linking blog.](../../../blog/posts/2024-08-15-bias-continued.md)\n",
    "\n",
    "This notebook will guide you through a 5-step process for evaluating bias in a data linking pipeline. It offers an isolated approach to **bias detection**, exploring potential mitigation options, before drawing conclusions about the existence of bias in the pipeline.\n",
    "\n",
    "This approach depends on users having pre-developed hypotheses about bias, based on a thorough understanding of the input data and model design. We’ll illustrate the approach with an example hypothesis, using Splink to show how each step can be practically applied.\n",
    "\n",
    "Consider this hypothesis:\n",
    "\n",
    "_I’m working with a 50,000-row dataset in which people appear multiple times over their lives. Because their information changes over time, records relating to the same individuals may not always be identical. I suspect that these changes could negatively impact linkage performance for those who update their details. Since women in this dataset are more likely to change their surname and postcode than men, this negative impact would disproportionately affect women, leading to a gender bias._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **1. Generate synthetic data**\n",
    "\n",
    "Since the hypothesis deals with changes in surnames and addresses, the synthetic data should include multiple records for the same person with these variations:\n",
    "\n",
    "The synthetic data should also match the structure and format of the production data used to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "synthetic_base_raw = [\n",
    "    {\"unique_id\": 1, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"brown\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"ba12 0ay\", \"gender\": \"female\", \"occupation\": \"politician\"}\n",
    "]\n",
    "\n",
    "synthetic_comparison_raw = [\n",
    "    {\"unique_id\": 2, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"brown\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"ba12 0ay\", \"gender\": \"female\", \"occupation\": \"politician\"},\n",
    "    {\"unique_id\": 3, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"doyle\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"ba12 0ay\", \"gender\": \"female\", \"occupation\": \"politician\"},\n",
    "    {\"unique_id\": 4, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"brown\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"tf3 2ng\", \"gender\": \"female\", \"occupation\": \"politician\"},\n",
    "    {\"unique_id\": 5, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"doyle\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"tf3 2ng\", \"gender\": \"female\", \"occupation\": \"politician\"},\n",
    "    {\"unique_id\": 6, \"person_id\": 2, \"first_name\": \"jane\", \"surname\": \"brown\", \"dob\": \"1860-01-01\", \"birth_place\": \"london\", \"postcode_fake\": \"ba12 0ay\", \"gender\": \"female\", \"occupation\": \"artist\"}\n",
    "]\n",
    "\n",
    "synthetic_base_df = pd.DataFrame(synthetic_base_raw)\n",
    "synthetic_comparison_df = pd.DataFrame(synthetic_comparison_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>person_id</th>\n",
       "      <th>first_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>dob</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>postcode_fake</th>\n",
       "      <th>gender</th>\n",
       "      <th>occupation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>sarah</td>\n",
       "      <td>brown</td>\n",
       "      <td>1862-07-11</td>\n",
       "      <td>london</td>\n",
       "      <td>ba12 0ay</td>\n",
       "      <td>female</td>\n",
       "      <td>politician</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id  person_id first_name surname         dob birth_place  \\\n",
       "0          1          1      sarah   brown  1862-07-11      london   \n",
       "\n",
       "  postcode_fake  gender  occupation  \n",
       "0      ba12 0ay  female  politician  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "synthetic_base_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unique_id</th>\n",
       "      <th>person_id</th>\n",
       "      <th>first_name</th>\n",
       "      <th>surname</th>\n",
       "      <th>dob</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>postcode_fake</th>\n",
       "      <th>gender</th>\n",
       "      <th>occupation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>sarah</td>\n",
       "      <td>brown</td>\n",
       "      <td>1862-07-11</td>\n",
       "      <td>london</td>\n",
       "      <td>ba12 0ay</td>\n",
       "      <td>female</td>\n",
       "      <td>politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>sarah</td>\n",
       "      <td>doyle</td>\n",
       "      <td>1862-07-11</td>\n",
       "      <td>london</td>\n",
       "      <td>ba12 0ay</td>\n",
       "      <td>female</td>\n",
       "      <td>politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>sarah</td>\n",
       "      <td>brown</td>\n",
       "      <td>1862-07-11</td>\n",
       "      <td>london</td>\n",
       "      <td>tf3 2ng</td>\n",
       "      <td>female</td>\n",
       "      <td>politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>sarah</td>\n",
       "      <td>doyle</td>\n",
       "      <td>1862-07-11</td>\n",
       "      <td>london</td>\n",
       "      <td>tf3 2ng</td>\n",
       "      <td>female</td>\n",
       "      <td>politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>jane</td>\n",
       "      <td>brown</td>\n",
       "      <td>1860-01-01</td>\n",
       "      <td>london</td>\n",
       "      <td>ba12 0ay</td>\n",
       "      <td>female</td>\n",
       "      <td>artist</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   unique_id  person_id first_name surname         dob birth_place  \\\n",
       "0          2          1      sarah   brown  1862-07-11      london   \n",
       "1          3          1      sarah   doyle  1862-07-11      london   \n",
       "2          4          1      sarah   brown  1862-07-11      london   \n",
       "3          5          1      sarah   doyle  1862-07-11      london   \n",
       "4          6          2       jane   brown  1860-01-01      london   \n",
       "\n",
       "  postcode_fake  gender  occupation  \n",
       "0      ba12 0ay  female  politician  \n",
       "1      ba12 0ay  female  politician  \n",
       "2       tf3 2ng  female  politician  \n",
       "3       tf3 2ng  female  politician  \n",
       "4      ba12 0ay  female      artist  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "synthetic_comparison_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first row is the base individual, with each subsequent row representing a different comparison related to the hypothesis. The first comparison is a self-link to serve as a baseline. Next, rows represent isolated surname changes and postcode changes, and finally a combination of both. \n",
    "\n",
    "Since this hypothesis relates to the linkage of records that _look different but are for the same person_, it’s helpful to include a record that _looks similar but is for a different person_ (like a sibling or partner). This will help account for the downstream effects of any mitigation option (Step 4).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **2. Train and investigate model**\n",
    "\n",
    "Train the model on the real production data (this Splink model is trained in 'model_create_h50k.ipynb' and saved as a json file).  \n",
    "You can refer to the [Splink tutorial](https://moj-analytical-services.github.io/splink/demos/tutorials/00_Tutorial_Introduction.html) if you need any further explanation on how to prepare the data and define the model settings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from splink import DuckDBAPI\n",
    "from splink import Linker, SettingsCreator\n",
    "from splink import splink_datasets\n",
    "\n",
    "db_api = DuckDBAPI()\n",
    "\n",
    "production_df = splink_datasets.historical_50k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "linker = Linker(production_df, settings='../../demo_settings/model_h50k.json', db_api=db_api)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's useful to visualise the model parameters to learn the relative importance of different parts of your data for linking.\n",
    "\n",
    "You can do this in Splink by using a match weights chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-a8fdd93c227445cc9077d0c6b27b6f41.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-a8fdd93c227445cc9077d0c6b27b6f41.vega-embed details,\n",
       "  #altair-viz-a8fdd93c227445cc9077d0c6b27b6f41.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-a8fdd93c227445cc9077d0c6b27b6f41\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-a8fdd93c227445cc9077d0c6b27b6f41\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-a8fdd93c227445cc9077d0c6b27b6f41\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300, \"discreteHeight\": 60, \"discreteWidth\": 400}, \"header\": {\"title\": null}, \"mark\": {\"tooltip\": null}, \"title\": {\"anchor\": \"middle\"}}, \"vconcat\": [{\"mark\": {\"type\": \"bar\", \"clip\": true, \"height\": 15}, \"encoding\": {\"color\": {\"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-10, 0, 10], \"interpolate\": \"lab\", \"range\": [\"red\", \"#bbbbbb\", \"green\"]}, \"title\": \"Match weight\", \"type\": \"quantitative\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"title\": \"Comparison name\", \"type\": \"nominal\"}, {\"field\": \"probability_two_random_records_match\", \"format\": \".4f\", \"title\": \"Probability two random records match\", \"type\": \"nominal\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Equivalent match weight\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"domain\": false, \"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}, \"labels\": false, \"ticks\": false, \"title\": \"\"}, \"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-13, 13]}, \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": \"Prior (starting) match weight\", \"titleAlign\": \"right\", \"titleAngle\": 0, \"titleFontWeight\": \"normal\"}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": 20, \"transform\": [{\"filter\": \"(datum.comparison_name == 'probability_two_random_records_match')\"}]}, {\"mark\": {\"type\": \"bar\", \"clip\": true}, \"encoding\": {\"color\": {\"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-10, 0, 10], \"interpolate\": \"lab\", \"range\": [\"red\", \"#bbbbbb\", \"green\"]}, \"title\": \"Match weight\", \"type\": \"quantitative\"}, \"row\": {\"field\": \"comparison_name\", \"header\": {\"labelAlign\": \"left\", \"labelAnchor\": \"middle\", \"labelAngle\": 0}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"title\": \"Comparison name\", \"type\": \"nominal\"}, {\"field\": \"label_for_charts\", \"title\": \"Label\", \"type\": \"ordinal\"}, {\"field\": \"sql_condition\", \"title\": \"SQL condition\", \"type\": \"nominal\"}, {\"field\": \"m_probability\", \"format\": \".10~g\", \"title\": \"M probability\", \"type\": \"quantitative\"}, {\"field\": \"u_probability\", \"format\": \".10~g\", \"title\": \"U probability\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor\", \"format\": \",.6f\", \"title\": \"Bayes factor = m/u\", \"type\": \"quantitative\"}, {\"field\": \"log2_bayes_factor\", \"format\": \".4~g\", \"title\": \"Match weight = log2(m/u)\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}, {\"field\": \"m_probability_description\", \"title\": \"m probability description\", \"type\": \"nominal\"}, {\"field\": \"u_probability_description\", \"title\": \"u probability description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}, \"title\": \"Comparison level match weight = log2(m/u)\"}, \"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-13, 13]}, \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": null}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": {\"step\": 12}, \"resolve\": {\"axis\": {\"y\": \"independent\"}, \"scale\": {\"y\": \"independent\"}}, \"transform\": [{\"filter\": \"(datum.comparison_name != 'probability_two_random_records_match')\"}]}], \"data\": {\"name\": \"data-867ce81f70e7312bfed110268f2e02f0\"}, \"params\": [{\"name\": \"mouse_zoom\", \"select\": {\"type\": \"interval\", \"encodings\": [\"x\"]}, \"bind\": \"scales\", \"views\": []}], \"resolve\": {\"axis\": {\"y\": \"independent\"}, \"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Model parameters (components of final match weight)\", \"subtitle\": \"Use mousewheel to zoom\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-867ce81f70e7312bfed110268f2e02f0\": [{\"comparison_name\": \"probability_two_random_records_match\", \"sql_condition\": null, \"label_for_charts\": \"\", \"m_probability\": null, \"u_probability\": null, \"m_probability_description\": null, \"u_probability_description\": null, \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": null, \"is_null_level\": false, \"bayes_factor\": 0.00013584539607096294, \"log2_bayes_factor\": -12.845746707461347, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 0, \"bayes_factor_description\": \"The probability that two random records drawn at random match is 0.000 or one in  7,362.3 records.This is equivalent to a starting match weight of -12.846.\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": -1}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.5457057743059716, \"u_probability\": 0.011951030823329812, \"m_probability_description\": \"Amongst matching record comparisons, 54.57% of records (i.e. one in 1.832) are in the exact match on first_name comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.195% of records (i.e. one in 83.67) are in the exact match on first_name comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"first_name\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 45.66181632137455, \"log2_bayes_factor\": 5.512916342661154, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 45.66 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.92\", \"m_probability\": 0.06098051520443211, \"u_probability\": 0.0018262673367931644, \"m_probability_description\": \"Amongst matching record comparisons, 6.098% of records (i.e. one in 16.4) are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.1826% of records (i.e. one in 548) are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 33.390793327942276, \"log2_bayes_factor\": 5.0613784655178495, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.92` then comparison is 33.39 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.88\", \"m_probability\": 0.07502792175772721, \"u_probability\": 0.003353844214458144, \"m_probability_description\": \"Amongst matching record comparisons, 7.503% of records (i.e. one in 13.33) are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.3354% of records (i.e. one in 298) are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 22.370723551883557, \"log2_bayes_factor\": 4.483540014128955, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.88` then comparison is 22.37 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.7\", \"m_probability\": 0.120309986619356, \"u_probability\": 0.020885448595733304, \"m_probability_description\": \"Amongst matching record comparisons, 12.03% of records (i.e. one in 8.312) are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 2.089% of records (i.e. one in 47.88) are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 5.760469355871733, \"log2_bayes_factor\": 2.5261863654525976, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.7` then comparison is 5.76 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.19797580211251312, \"u_probability\": 0.9619834090296856, \"m_probability_description\": \"Amongst matching record comparisons, 19.8% of records (i.e. one in 5.051) are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 96.2% of records (i.e. one in 1.04) are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.20579960138003153, \"log2_bayes_factor\": -2.2806879070559116, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 4.859 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 0}, {\"comparison_name\": \"surname\", \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.7714400787402513, \"u_probability\": 0.0006788109515158729, \"m_probability_description\": \"Amongst matching record comparisons, 77.14% of records (i.e. one in 1.296) are in the exact match on surname comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.06788% of records (i.e. one in 1,473) are in the exact match on surname comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"surname\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 1136.4579151493144, \"log2_bayes_factor\": 10.150328544506749, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 1,136 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.92\", \"m_probability\": 0.09368190481882498, \"u_probability\": 0.0002900130152182439, \"m_probability_description\": \"Amongst matching record comparisons, 9.368% of records (i.e. one in 10.67) are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.029% of records (i.e. one in 3,448) are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 323.02655364734716, \"log2_bayes_factor\": 8.335508952962527, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.92` then comparison is 323 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.88\", \"m_probability\": 0.039136395099805245, \"u_probability\": 0.00048319241644015245, \"m_probability_description\": \"Amongst matching record comparisons, 3.914% of records (i.e. one in 25.55) are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.04832% of records (i.e. one in 2,070) are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 80.99546633644782, \"log2_bayes_factor\": 6.339769251317024, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.88` then comparison is 81 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.7\", \"m_probability\": 0.023569512976655334, \"u_probability\": 0.017232968503952374, \"m_probability_description\": \"Amongst matching record comparisons, 2.357% of records (i.e. one in 42.43) are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.723% of records (i.e. one in 58.03) are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 1.3676989528095334, \"log2_bayes_factor\": 0.4517507104250282, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.7` then comparison is 1.368 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.07217210836446314, \"u_probability\": 0.9813150151128733, \"m_probability_description\": \"Amongst matching record comparisons, 7.217% of records (i.e. one in 13.86) are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 98.13% of records (i.e. one in 1.019) are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.07354632024677796, \"log2_bayes_factor\": -3.765203029049615, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 13.6 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 1}, {\"comparison_name\": \"dob\", \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on date of birth\", \"m_probability\": 0.6796640528638345, \"u_probability\": 0.0023542362733926883, \"m_probability_description\": \"Amongst matching record comparisons, 67.97% of records (i.e. one in 1.471) are in the exact match on date of birth comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.2354% of records (i.e. one in 425) are in the exact match on date of birth comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 288.6983182381992, \"log2_bayes_factor\": 8.173418893204259, \"comparison_vector_value\": 5, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `exact match on date of birth` then comparison is 289 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"damerau_levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"label_for_charts\": \"DamerauLevenshtein distance <= 1\", \"m_probability\": 0.2736730864202431, \"u_probability\": 0.02425500685606134, \"m_probability_description\": \"Amongst matching record comparisons, 27.37% of records (i.e. one in 3.654) are in the dameraulevenshtein distance <= 1 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 2.426% of records (i.e. one in 41.23) are in the dameraulevenshtein distance <= 1 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 11.28315848535215, \"log2_bayes_factor\": 3.4960990716057827, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `dameraulevenshtein distance <= 1` then comparison is 11.28 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 2629800.0\", \"label_for_charts\": \"Abs date difference <= 1 month\", \"m_probability\": 0.002677966995147039, \"u_probability\": 0.0023411035361826224, \"m_probability_description\": \"Amongst matching record comparisons, 0.2678% of records (i.e. one in 373) are in the abs date difference <= 1 month comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.2341% of records (i.e. one in 427) are in the abs date difference <= 1 month comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 1.143890884686673, \"log2_bayes_factor\": 0.19394944055222407, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `abs date difference <= 1 month` then comparison is 1.144 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 31557600.0\", \"label_for_charts\": \"Abs date difference <= 1 year\", \"m_probability\": 0.006199700352635373, \"u_probability\": 0.03354178334846173, \"m_probability_description\": \"Amongst matching record comparisons, 0.62% of records (i.e. one in 161) are in the abs date difference <= 1 year comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 3.354% of records (i.e. one in 29.81) are in the abs date difference <= 1 year comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.1848351439226531, \"log2_bayes_factor\": -2.4356890029468414, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `abs date difference <= 1 year` then comparison is 5.41 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 315576000.0\", \"label_for_charts\": \"Abs date difference <= 10 year\", \"m_probability\": 0.02726057331431288, \"u_probability\": 0.25042855211572257, \"m_probability_description\": \"Amongst matching record comparisons, 2.726% of records (i.e. one in 36.68) are in the abs date difference <= 10 year comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 25.04% of records (i.e. one in 3.993) are in the abs date difference <= 10 year comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.10885569190894744, \"log2_bayes_factor\": -3.1995112489219304, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `abs date difference <= 10 year` then comparison is 9.186 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.010524620053827111, \"u_probability\": 0.6870793178701791, \"m_probability_description\": \"Amongst matching record comparisons, 1.052% of records (i.e. one in 95.02) are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 68.71% of records (i.e. one in 1.455) are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.01531791130964547, \"log2_bayes_factor\": -6.028636599342855, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 65.28 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 2}, {\"comparison_name\": \"postcode_fake\", \"sql_condition\": \"\\\"postcode_fake_l\\\" = \\\"postcode_fake_r\\\"\", \"label_for_charts\": \"Exact match on full postcode\", \"m_probability\": 0.6753757141159644, \"u_probability\": 0.00014903079407788038, \"m_probability_description\": \"Amongst matching record comparisons, 67.54% of records (i.e. one in 1.481) are in the exact match on full postcode comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.0149% of records (i.e. one in 6,710) are in the exact match on full postcode comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 4531.786321711654, \"log2_bayes_factor\": 12.14586412290028, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on full postcode` then comparison is 4,532 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 3}, {\"comparison_name\": \"postcode_fake\", \"sql_condition\": \"NULLIF(regexp_extract(\\\"postcode_fake_l\\\", '^[A-Za-z]{1,2}[0-9][A-Za-z0-9]? [0-9]', 0), '') = NULLIF(regexp_extract(\\\"postcode_fake_r\\\", '^[A-Za-z]{1,2}[0-9][A-Za-z0-9]? [0-9]', 0), '')\", \"label_for_charts\": \"Exact match on sector\", \"m_probability\": 0.09395110908506221, \"u_probability\": 0.00028996941834157726, \"m_probability_description\": \"Amongst matching record comparisons, 9.395% of records (i.e. one in 10.64) are in the exact match on sector comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.029% of records (i.e. one in 3,449) are in the exact match on sector comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 324.00350913691864, \"log2_bayes_factor\": 8.33986562815319, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on sector` then comparison is 324 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 3}, {\"comparison_name\": \"postcode_fake\", \"sql_condition\": \"NULLIF(regexp_extract(\\\"postcode_fake_l\\\", '^[A-Za-z]{1,2}[0-9][A-Za-z0-9]?', 0), '') = NULLIF(regexp_extract(\\\"postcode_fake_r\\\", '^[A-Za-z]{1,2}[0-9][A-Za-z0-9]?', 0), '')\", \"label_for_charts\": \"Exact match on district\", \"m_probability\": 0.04225883501370074, \"u_probability\": 0.00048215845142843657, \"m_probability_description\": \"Amongst matching record comparisons, 4.226% of records (i.e. one in 23.66) are in the exact match on district comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.04822% of records (i.e. one in 2,074) are in the exact match on district comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 87.64511933474411, \"log2_bayes_factor\": 6.453601849219456, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on district` then comparison is 87.65 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 3}, {\"comparison_name\": \"postcode_fake\", \"sql_condition\": \"NULLIF(regexp_extract(\\\"postcode_fake_l\\\", '^[A-Za-z]{1,2}', 0), '') = NULLIF(regexp_extract(\\\"postcode_fake_r\\\", '^[A-Za-z]{1,2}', 0), '')\", \"label_for_charts\": \"Exact match on area\", \"m_probability\": 0.09785854822786318, \"u_probability\": 0.011052555271205699, \"m_probability_description\": \"Amongst matching record comparisons, 9.786% of records (i.e. one in 10.22) are in the exact match on area comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.105% of records (i.e. one in 90.48) are in the exact match on area comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 8.853929777017798, \"log2_bayes_factor\": 3.1463179310651057, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on area` then comparison is 8.854 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 3}, {\"comparison_name\": \"postcode_fake\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.09055579355740942, \"u_probability\": 0.9880262860649464, \"m_probability_description\": \"Amongst matching record comparisons, 9.056% of records (i.e. one in 11.04) are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 98.8% of records (i.e. one in 1.012) are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.0916532230312108, \"log2_bayes_factor\": -3.4476705750282615, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 10.91 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 3}, {\"comparison_name\": \"birth_place\", \"sql_condition\": \"\\\"birth_place_l\\\" = \\\"birth_place_r\\\"\", \"label_for_charts\": \"Exact match on birth_place\", \"m_probability\": 0.8345399212327826, \"u_probability\": 0.005228068621252938, \"m_probability_description\": \"Amongst matching record comparisons, 83.45% of records (i.e. one in 1.198) are in the exact match on birth_place comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.5228% of records (i.e. one in 191) are in the exact match on birth_place comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 159.62681091067626, \"log2_bayes_factor\": 7.318559176669256, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on birth_place` then comparison is 160 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 4}, {\"comparison_name\": \"birth_place\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.1654600787672175, \"u_probability\": 0.9947719313787471, \"m_probability_description\": \"Amongst matching record comparisons, 16.55% of records (i.e. one in 6.044) are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 99.48% of records (i.e. one in 1.005) are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.16632966165208438, \"log2_bayes_factor\": -2.587882626788239, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 6.012 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 4}, {\"comparison_name\": \"occupation\", \"sql_condition\": \"\\\"occupation_l\\\" = \\\"occupation_r\\\"\", \"label_for_charts\": \"Exact match on occupation\", \"m_probability\": 0.8982446653774181, \"u_probability\": 0.037284666827814034, \"m_probability_description\": \"Amongst matching record comparisons, 89.82% of records (i.e. one in 1.113) are in the exact match on occupation comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 3.728% of records (i.e. one in 26.82) are in the exact match on occupation comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 24.09152989151389, \"log2_bayes_factor\": 4.590454107260654, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on occupation` then comparison is 24.09 times more likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 5}, {\"comparison_name\": \"occupation\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.10175533462258204, \"u_probability\": 0.9627153331721859, \"m_probability_description\": \"Amongst matching record comparisons, 10.18% of records (i.e. one in 9.827) are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 96.27% of records (i.e. one in 1.039) are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.10569618153612875, \"log2_bayes_factor\": -3.2420048371677, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 9.461 times less likely to be a match\", \"probability_two_random_records_match\": 0.00013582694460587586, \"comparison_sort_order\": 5}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.VConcatChart(...)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linker.visualisations.match_weights_chart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When developing this model, it's essential to evaluate the parameter weights to ensure they align with the pipeline's goals. However, this approach requires considering the bias hypothesis.\n",
    "\n",
    "At first glance, the model appears normal, but a few key points stand out regarding the hypothesis (focussed on surname and postcode changes):\n",
    "\n",
    "- A non-match on the surname is more predictive than a non-match on the first name.\n",
    "- A postcode match has the highest predictive weight.\n",
    "\n",
    "The cumulative effect of these weights makes it difficult to fully understand their impact without generating comparisons.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **3. Perform and evaluate linkage**\n",
    "\n",
    "In a standard linkage with Splink, all records which meet the criteria of the blocking rules would be compared against each other. However, in bias detection we may not be interested in a lot of the comparisons this process will generate match probabilities for. It can be easier to manually generate the comparisons that are relevant to the hypothesis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_records(base_records, comparison_records, linker):\n",
    "    results = []\n",
    "    for record_1 in base_records:\n",
    "        results.extend(\n",
    "            linker.inference.compare_two_records(record_1, record_2).as_pandas_dataframe()\n",
    "            for record_2 in comparison_records\n",
    "        )\n",
    "    all_comparisons_df = pd.concat(results, ignore_index=True)\n",
    "    return all_comparisons_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n"
     ]
    }
   ],
   "source": [
    "comparisons = compare_records(synthetic_base_raw, synthetic_comparison_raw, linker)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the resulting match probabilities in terms of the threshold of your pipeline.  \n",
    "This pipeline has a high match threshold of 0.999, medium of 0.99, and low of 0.95."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def highlight_cells(val):\n",
    "    if val >= 0.999:\n",
    "        color = '#c4f5bf'  # High threshold, green\n",
    "    elif val >= 0.99:\n",
    "        color = '#faf9c0'  # Medium threshold, yellow\n",
    "    elif val >= 0.95:\n",
    "        color = '#f5e1bf'  # Low threshold, orange\n",
    "    else:\n",
    "        color = '#f5c8bf'  # Below threshold, red\n",
    "    return f'background-color: {color}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_744a8_row0_col1, #T_744a8_row1_col1, #T_744a8_row2_col1 {\n",
       "  background-color: #c4f5bf;\n",
       "}\n",
       "#T_744a8_row3_col1 {\n",
       "  background-color: #f5e1bf;\n",
       "}\n",
       "#T_744a8_row4_col1 {\n",
       "  background-color: #faf9c0;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_744a8\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_744a8_level0_col0\" class=\"col_heading level0 col0\" >match_weight</th>\n",
       "      <th id=\"T_744a8_level0_col1\" class=\"col_heading level0 col1\" >match_probability</th>\n",
       "      <th id=\"T_744a8_level0_col2\" class=\"col_heading level0 col2\" >unique_id_l</th>\n",
       "      <th id=\"T_744a8_level0_col3\" class=\"col_heading level0 col3\" >unique_id_r</th>\n",
       "      <th id=\"T_744a8_level0_col4\" class=\"col_heading level0 col4\" >first_name_l</th>\n",
       "      <th id=\"T_744a8_level0_col5\" class=\"col_heading level0 col5\" >first_name_r</th>\n",
       "      <th id=\"T_744a8_level0_col6\" class=\"col_heading level0 col6\" >surname_l</th>\n",
       "      <th id=\"T_744a8_level0_col7\" class=\"col_heading level0 col7\" >surname_r</th>\n",
       "      <th id=\"T_744a8_level0_col8\" class=\"col_heading level0 col8\" >dob_l</th>\n",
       "      <th id=\"T_744a8_level0_col9\" class=\"col_heading level0 col9\" >dob_r</th>\n",
       "      <th id=\"T_744a8_level0_col10\" class=\"col_heading level0 col10\" >postcode_fake_l</th>\n",
       "      <th id=\"T_744a8_level0_col11\" class=\"col_heading level0 col11\" >postcode_fake_r</th>\n",
       "      <th id=\"T_744a8_level0_col12\" class=\"col_heading level0 col12\" >birth_place_l</th>\n",
       "      <th id=\"T_744a8_level0_col13\" class=\"col_heading level0 col13\" >birth_place_r</th>\n",
       "      <th id=\"T_744a8_level0_col14\" class=\"col_heading level0 col14\" >occupation_l</th>\n",
       "      <th id=\"T_744a8_level0_col15\" class=\"col_heading level0 col15\" >occupation_r</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_744a8_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_744a8_row0_col0\" class=\"data row0 col0\" >35.045794</td>\n",
       "      <td id=\"T_744a8_row0_col1\" class=\"data row0 col1\" >1.000000</td>\n",
       "      <td id=\"T_744a8_row0_col2\" class=\"data row0 col2\" >1</td>\n",
       "      <td id=\"T_744a8_row0_col3\" class=\"data row0 col3\" >2</td>\n",
       "      <td id=\"T_744a8_row0_col4\" class=\"data row0 col4\" >sarah</td>\n",
       "      <td id=\"T_744a8_row0_col5\" class=\"data row0 col5\" >sarah</td>\n",
       "      <td id=\"T_744a8_row0_col6\" class=\"data row0 col6\" >brown</td>\n",
       "      <td id=\"T_744a8_row0_col7\" class=\"data row0 col7\" >brown</td>\n",
       "      <td id=\"T_744a8_row0_col8\" class=\"data row0 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row0_col9\" class=\"data row0 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row0_col10\" class=\"data row0 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row0_col11\" class=\"data row0 col11\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row0_col12\" class=\"data row0 col12\" >london</td>\n",
       "      <td id=\"T_744a8_row0_col13\" class=\"data row0 col13\" >london</td>\n",
       "      <td id=\"T_744a8_row0_col14\" class=\"data row0 col14\" >politician</td>\n",
       "      <td id=\"T_744a8_row0_col15\" class=\"data row0 col15\" >politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_744a8_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_744a8_row1_col0\" class=\"data row1 col0\" >21.130263</td>\n",
       "      <td id=\"T_744a8_row1_col1\" class=\"data row1 col1\" >1.000000</td>\n",
       "      <td id=\"T_744a8_row1_col2\" class=\"data row1 col2\" >1</td>\n",
       "      <td id=\"T_744a8_row1_col3\" class=\"data row1 col3\" >3</td>\n",
       "      <td id=\"T_744a8_row1_col4\" class=\"data row1 col4\" >sarah</td>\n",
       "      <td id=\"T_744a8_row1_col5\" class=\"data row1 col5\" >sarah</td>\n",
       "      <td id=\"T_744a8_row1_col6\" class=\"data row1 col6\" >brown</td>\n",
       "      <td id=\"T_744a8_row1_col7\" class=\"data row1 col7\" >doyle</td>\n",
       "      <td id=\"T_744a8_row1_col8\" class=\"data row1 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row1_col9\" class=\"data row1 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row1_col10\" class=\"data row1 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row1_col11\" class=\"data row1 col11\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row1_col12\" class=\"data row1 col12\" >london</td>\n",
       "      <td id=\"T_744a8_row1_col13\" class=\"data row1 col13\" >london</td>\n",
       "      <td id=\"T_744a8_row1_col14\" class=\"data row1 col14\" >politician</td>\n",
       "      <td id=\"T_744a8_row1_col15\" class=\"data row1 col15\" >politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_744a8_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_744a8_row2_col0\" class=\"data row2 col0\" >19.452260</td>\n",
       "      <td id=\"T_744a8_row2_col1\" class=\"data row2 col1\" >0.999999</td>\n",
       "      <td id=\"T_744a8_row2_col2\" class=\"data row2 col2\" >1</td>\n",
       "      <td id=\"T_744a8_row2_col3\" class=\"data row2 col3\" >4</td>\n",
       "      <td id=\"T_744a8_row2_col4\" class=\"data row2 col4\" >sarah</td>\n",
       "      <td id=\"T_744a8_row2_col5\" class=\"data row2 col5\" >sarah</td>\n",
       "      <td id=\"T_744a8_row2_col6\" class=\"data row2 col6\" >brown</td>\n",
       "      <td id=\"T_744a8_row2_col7\" class=\"data row2 col7\" >brown</td>\n",
       "      <td id=\"T_744a8_row2_col8\" class=\"data row2 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row2_col9\" class=\"data row2 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row2_col10\" class=\"data row2 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row2_col11\" class=\"data row2 col11\" >tf3 2ng</td>\n",
       "      <td id=\"T_744a8_row2_col12\" class=\"data row2 col12\" >london</td>\n",
       "      <td id=\"T_744a8_row2_col13\" class=\"data row2 col13\" >london</td>\n",
       "      <td id=\"T_744a8_row2_col14\" class=\"data row2 col14\" >politician</td>\n",
       "      <td id=\"T_744a8_row2_col15\" class=\"data row2 col15\" >politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_744a8_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_744a8_row3_col0\" class=\"data row3 col0\" >5.536728</td>\n",
       "      <td id=\"T_744a8_row3_col1\" class=\"data row3 col1\" >0.978913</td>\n",
       "      <td id=\"T_744a8_row3_col2\" class=\"data row3 col2\" >1</td>\n",
       "      <td id=\"T_744a8_row3_col3\" class=\"data row3 col3\" >5</td>\n",
       "      <td id=\"T_744a8_row3_col4\" class=\"data row3 col4\" >sarah</td>\n",
       "      <td id=\"T_744a8_row3_col5\" class=\"data row3 col5\" >sarah</td>\n",
       "      <td id=\"T_744a8_row3_col6\" class=\"data row3 col6\" >brown</td>\n",
       "      <td id=\"T_744a8_row3_col7\" class=\"data row3 col7\" >doyle</td>\n",
       "      <td id=\"T_744a8_row3_col8\" class=\"data row3 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row3_col9\" class=\"data row3 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row3_col10\" class=\"data row3 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row3_col11\" class=\"data row3 col11\" >tf3 2ng</td>\n",
       "      <td id=\"T_744a8_row3_col12\" class=\"data row3 col12\" >london</td>\n",
       "      <td id=\"T_744a8_row3_col13\" class=\"data row3 col13\" >london</td>\n",
       "      <td id=\"T_744a8_row3_col14\" class=\"data row3 col14\" >politician</td>\n",
       "      <td id=\"T_744a8_row3_col15\" class=\"data row3 col15\" >politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_744a8_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_744a8_row4_col0\" class=\"data row4 col0\" >8.046801</td>\n",
       "      <td id=\"T_744a8_row4_col1\" class=\"data row4 col1\" >0.996233</td>\n",
       "      <td id=\"T_744a8_row4_col2\" class=\"data row4 col2\" >1</td>\n",
       "      <td id=\"T_744a8_row4_col3\" class=\"data row4 col3\" >6</td>\n",
       "      <td id=\"T_744a8_row4_col4\" class=\"data row4 col4\" >sarah</td>\n",
       "      <td id=\"T_744a8_row4_col5\" class=\"data row4 col5\" >jane</td>\n",
       "      <td id=\"T_744a8_row4_col6\" class=\"data row4 col6\" >brown</td>\n",
       "      <td id=\"T_744a8_row4_col7\" class=\"data row4 col7\" >brown</td>\n",
       "      <td id=\"T_744a8_row4_col8\" class=\"data row4 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_744a8_row4_col9\" class=\"data row4 col9\" >1860-01-01</td>\n",
       "      <td id=\"T_744a8_row4_col10\" class=\"data row4 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row4_col11\" class=\"data row4 col11\" >ba12 0ay</td>\n",
       "      <td id=\"T_744a8_row4_col12\" class=\"data row4 col12\" >london</td>\n",
       "      <td id=\"T_744a8_row4_col13\" class=\"data row4 col13\" >london</td>\n",
       "      <td id=\"T_744a8_row4_col14\" class=\"data row4 col14\" >politician</td>\n",
       "      <td id=\"T_744a8_row4_col15\" class=\"data row4 col15\" >artist</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f8875b3d8e0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns_of_interest = ['match_weight', 'match_probability', 'unique_id_l', 'unique_id_r', 'first_name_l', 'first_name_r', 'surname_l','surname_r', 'dob_l', 'dob_r','postcode_fake_l', 'postcode_fake_r','birth_place_l', 'birth_place_r', 'occupation_l', 'occupation_r']\n",
    "comparisons[columns_of_interest].style.map(highlight_cells, subset=['match_probability'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first three comparisons show high match probabilities. A surname change or postcode change alone allow for a linkage at a high match threshold.  \n",
    "However, when combined they lower the match probability to 0.9781 - which could only be linked at a low threshold in the example pipeline. \n",
    "\n",
    "Additionally, it is useful to note that the sibling scenario has a a higher match probability than the surname and postcode change scenario, and it can be matched at a medium threshold. \n",
    "\n",
    "It’s helpful to break these results down further to understand how the individual model parameters are combining to result in these final probabilities. This can be done in Splink using the waterfall chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-eacdee7467a14e0aa2835d6c6530ded6.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-eacdee7467a14e0aa2835d6c6530ded6.vega-embed details,\n",
       "  #altair-viz-eacdee7467a14e0aa2835d6c6530ded6.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-eacdee7467a14e0aa2835d6c6530ded6\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-eacdee7467a14e0aa2835d6c6530ded6\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-eacdee7467a14e0aa2835d6c6530ded6\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.16.3?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.16.3\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"layer\": [{\"layer\": [{\"mark\": \"rule\", \"encoding\": {\"color\": {\"value\": \"black\"}, \"size\": {\"value\": 0.5}, \"y\": {\"field\": \"zero\", \"type\": \"quantitative\"}}}, {\"mark\": {\"type\": \"bar\", \"width\": 60}, \"encoding\": {\"color\": {\"condition\": {\"test\": \"(datum.log2_bayes_factor < 0)\", \"value\": \"red\"}, \"value\": \"green\"}, \"opacity\": {\"condition\": {\"test\": \"datum.column_name == 'Prior match weight' || datum.column_name == 'Final score'\", \"value\": 1}, \"value\": 0.5}, \"tooltip\": [{\"field\": \"column_name\", \"title\": \"Comparison column\", \"type\": \"nominal\"}, {\"field\": \"value_l\", \"title\": \"Value (L)\", \"type\": \"nominal\"}, {\"field\": \"value_r\", \"title\": \"Value (R)\", \"type\": \"nominal\"}, {\"field\": \"label_for_charts\", \"title\": \"Label\", \"type\": \"ordinal\"}, {\"field\": \"sql_condition\", \"title\": \"SQL condition\", \"type\": \"nominal\"}, {\"field\": \"comparison_vector_value\", \"title\": \"Comparison vector value\", \"type\": \"nominal\"}, {\"field\": \"bayes_factor\", \"format\": \",.4f\", \"title\": \"Bayes factor = m/u\", \"type\": \"quantitative\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Match weight = log2(m/u)\", \"type\": \"quantitative\"}, {\"field\": \"prob\", \"format\": \".4f\", \"title\": \"Cumulative match probability\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"grid\": true, \"labelAlign\": \"center\", \"labelAngle\": -20, \"labelExpr\": \"datum.value == 'Prior' || datum.value == 'Final score' ? '' : datum.value\", \"labelPadding\": 10, \"tickBand\": \"extent\", \"title\": \"Column\"}, \"field\": \"column_name\", \"sort\": {\"field\": \"bar_sort_order\", \"order\": \"ascending\"}, \"type\": \"nominal\"}, \"y\": {\"axis\": {\"grid\": false, \"orient\": \"left\", \"title\": \"Match Weight\"}, \"field\": \"previous_sum\", \"type\": \"quantitative\"}, \"y2\": {\"field\": \"sum\"}}}, {\"mark\": {\"type\": \"text\", \"fontWeight\": \"bold\"}, \"encoding\": {\"color\": {\"value\": \"white\"}, \"text\": {\"condition\": {\"test\": \"abs(datum.log2_bayes_factor) > 1\", \"field\": \"log2_bayes_factor\", \"format\": \".2f\", \"type\": \"nominal\"}, \"value\": \"\"}, \"x\": {\"axis\": {\"labelAngle\": -20, \"title\": \"Column\"}, \"field\": \"column_name\", \"sort\": {\"field\": \"bar_sort_order\", \"order\": \"ascending\"}, \"type\": \"nominal\"}, \"y\": {\"axis\": {\"orient\": \"left\"}, \"field\": \"center\", \"type\": \"quantitative\"}}}, {\"mark\": {\"type\": \"text\", \"baseline\": \"bottom\", \"dy\": -25, \"fontWeight\": \"bold\"}, \"encoding\": {\"color\": {\"value\": \"black\"}, \"text\": {\"field\": \"column_name\", \"type\": \"nominal\"}, \"x\": {\"axis\": {\"labelAngle\": -20, \"title\": \"Column\"}, \"field\": \"column_name\", \"sort\": {\"field\": \"bar_sort_order\", \"order\": \"ascending\"}, \"type\": \"nominal\"}, \"y\": {\"field\": \"sum_top\", \"type\": \"quantitative\"}}}, {\"mark\": {\"type\": \"text\", \"baseline\": \"bottom\", \"dy\": -13, \"fontSize\": 8}, \"encoding\": {\"color\": {\"value\": \"grey\"}, \"text\": {\"field\": \"value_l\", \"type\": \"nominal\"}, \"x\": {\"axis\": {\"labelAngle\": -20, \"title\": \"Column\"}, \"field\": \"column_name\", \"sort\": {\"field\": \"bar_sort_order\", \"order\": \"ascending\"}, \"type\": \"nominal\"}, \"y\": {\"field\": \"sum_top\", \"type\": \"quantitative\"}}}, {\"mark\": {\"type\": \"text\", \"baseline\": \"bottom\", \"dy\": -5, \"fontSize\": 8}, \"encoding\": {\"color\": {\"value\": \"grey\"}, \"text\": {\"field\": \"value_r\", \"type\": \"nominal\"}, \"x\": {\"axis\": {\"labelAngle\": -20, \"title\": \"Column\"}, \"field\": \"column_name\", \"sort\": {\"field\": \"bar_sort_order\", \"order\": \"ascending\"}, \"type\": \"nominal\"}, \"y\": {\"field\": \"sum_top\", \"type\": \"quantitative\"}}}]}, {\"mark\": {\"type\": \"rule\", \"color\": \"black\", \"strokeWidth\": 2, \"x2Offset\": 30, \"xOffset\": -30}, \"encoding\": {\"x\": {\"axis\": {\"labelAngle\": -20, \"title\": \"Column\"}, \"field\": \"column_name\", \"sort\": {\"field\": \"bar_sort_order\", \"order\": \"ascending\"}, \"type\": \"nominal\"}, \"x2\": {\"field\": \"lead\"}, \"y\": {\"axis\": {\"labelExpr\": \"format(1 / (1 + pow(2, -1*datum.value)), '.2r')\", \"orient\": \"right\", \"title\": \"Probability\"}, \"field\": \"sum\", \"scale\": {\"zero\": false}, \"type\": \"quantitative\"}}}], \"data\": {\"name\": \"data-188596eaf73ce26accc4a99af0059c50\"}, \"height\": 450, \"params\": [{\"name\": \"record_number\", \"bind\": {\"input\": \"range\", \"max\": 4, \"min\": 0, \"step\": 1}, \"value\": 0}], \"resolve\": {\"axis\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Match weights waterfall chart\", \"subtitle\": \"How each comparison contributes to the final match score\"}, \"transform\": [{\"filter\": \"(datum.record_number == record_number)\"}, {\"filter\": \"(datum.bayes_factor !== 1.0)\"}, {\"window\": [{\"op\": \"sum\", \"field\": \"log2_bayes_factor\", \"as\": \"sum\"}, {\"op\": \"lead\", \"field\": \"column_name\", \"as\": \"lead\"}], \"frame\": [null, 0]}, {\"calculate\": \"datum.column_name === \\\"Final score\\\" ? datum.sum - datum.log2_bayes_factor : datum.sum\", \"as\": \"sum\"}, {\"calculate\": \"datum.lead === null ? datum.column_name : datum.lead\", \"as\": \"lead\"}, {\"calculate\": \"datum.column_name === \\\"Final score\\\" || datum.column_name === \\\"Prior match weight\\\" ? 0 : datum.sum - datum.log2_bayes_factor\", \"as\": \"previous_sum\"}, {\"calculate\": \"datum.sum > datum.previous_sum ? datum.column_name : \\\"\\\"\", \"as\": \"top_label\"}, {\"calculate\": \"datum.sum < datum.previous_sum ? datum.column_name : \\\"\\\"\", \"as\": \"bottom_label\"}, {\"calculate\": \"datum.sum > datum.previous_sum ? datum.sum : datum.previous_sum\", \"as\": \"sum_top\"}, {\"calculate\": \"datum.sum < datum.previous_sum ? datum.sum : datum.previous_sum\", \"as\": \"sum_bottom\"}, {\"calculate\": \"(datum.sum + datum.previous_sum) / 2\", \"as\": \"center\"}, {\"calculate\": \"(datum.log2_bayes_factor > 0 ? \\\"+\\\" : \\\"\\\") + datum.log2_bayes_factor\", \"as\": \"text_log2_bayes_factor\"}, {\"calculate\": \"datum.sum < datum.previous_sum ? 4 : -4\", \"as\": \"dy\"}, {\"calculate\": \"datum.sum < datum.previous_sum ? \\\"top\\\" : \\\"bottom\\\"\", \"as\": \"baseline\"}, {\"calculate\": \"1. / (1 + pow(2, -1.*datum.sum))\", \"as\": \"prob\"}, {\"calculate\": \"0*datum.sum\", \"as\": \"zero\"}], \"width\": {\"step\": 75}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-188596eaf73ce26accc4a99af0059c50\": [{\"column_name\": \"Prior\", \"label_for_charts\": \"Starting match weight (prior)\", \"sql_condition\": null, \"log2_bayes_factor\": -12.845746707461347, \"bayes_factor\": 0.00013584539607096294, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 0, \"record_number\": 0}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.5457057743059716, \"u_probability\": 0.011951030823329812, \"bayes_factor\": 45.66181632137455, \"log2_bayes_factor\": 5.512916342661154, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 45.66 times more likely to be a match\", \"column_name\": \"first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 1, \"record_number\": 0}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Term freq adjustment on first_name with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on first_name makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 2, \"record_number\": 0}, {\"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.7714400787402513, \"u_probability\": 0.0006788109515158729, \"bayes_factor\": 1136.4579151493144, \"log2_bayes_factor\": 10.150328544506749, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 1,136 times more likely to be a match\", \"column_name\": \"surname\", \"value_l\": \"brown\", \"value_r\": \"brown\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 3, \"record_number\": 0}, {\"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Term freq adjustment on surname with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on surname makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_surname\", \"value_l\": \"brown\", \"value_r\": \"brown\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 4, \"record_number\": 0}, {\"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on date of birth\", \"m_probability\": 0.6796640528638345, \"u_probability\": 0.0023542362733926883, \"bayes_factor\": 288.6983182381992, \"log2_bayes_factor\": 8.173418893204259, \"comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `exact match on date of birth` then comparison is 289 times more likely to be a match\", \"column_name\": \"dob\", \"value_l\": \"1862-07-11\", \"value_r\": \"1862-07-11\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 5, \"record_number\": 0}, {\"sql_condition\": \"\\\"postcode_fake_l\\\" = \\\"postcode_fake_r\\\"\", \"label_for_charts\": \"Exact match on full postcode\", \"m_probability\": 0.6753757141159644, \"u_probability\": 0.00014903079407788038, \"bayes_factor\": 4531.786321711654, \"log2_bayes_factor\": 12.14586412290028, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on full postcode` then comparison is 4,532 times more likely to be a match\", \"column_name\": \"postcode_fake\", \"value_l\": \"ba12 0ay\", \"value_r\": \"ba12 0ay\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 6, \"record_number\": 0}, {\"sql_condition\": \"\\\"birth_place_l\\\" = \\\"birth_place_r\\\"\", \"label_for_charts\": \"Exact match on birth_place\", \"m_probability\": 0.8345399212327826, \"u_probability\": 0.005228068621252938, \"bayes_factor\": 159.62681091067626, \"log2_bayes_factor\": 7.318559176669256, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on birth_place` then comparison is 160 times more likely to be a match\", \"column_name\": \"birth_place\", \"value_l\": \"london\", \"value_r\": \"london\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 7, \"record_number\": 0}, {\"sql_condition\": \"\\\"occupation_l\\\" = \\\"occupation_r\\\"\", \"label_for_charts\": \"Exact match on occupation\", \"m_probability\": 0.8982446653774181, \"u_probability\": 0.037284666827814034, \"bayes_factor\": 24.09152989151389, \"log2_bayes_factor\": 4.590454107260654, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on occupation` then comparison is 24.09 times more likely to be a match\", \"column_name\": \"occupation\", \"value_l\": \"politician\", \"value_r\": \"politician\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 8, \"record_number\": 0}, {\"column_name\": \"Final score\", \"label_for_charts\": \"Final score\", \"sql_condition\": null, \"log2_bayes_factor\": 35.045794479741005, \"bayes_factor\": 35467890604.4997, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 9, \"record_number\": 0}, {\"column_name\": \"Prior\", \"label_for_charts\": \"Starting match weight (prior)\", \"sql_condition\": null, \"log2_bayes_factor\": -12.845746707461347, \"bayes_factor\": 0.00013584539607096294, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 0, \"record_number\": 1}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.5457057743059716, \"u_probability\": 0.011951030823329812, \"bayes_factor\": 45.66181632137455, \"log2_bayes_factor\": 5.512916342661154, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 45.66 times more likely to be a match\", \"column_name\": \"first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 1, \"record_number\": 1}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Term freq adjustment on first_name with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on first_name makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 2, \"record_number\": 1}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.07217210836446314, \"u_probability\": 0.9813150151128733, \"bayes_factor\": 0.07354632024677796, \"log2_bayes_factor\": -3.765203029049615, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 13.6 times less likely to be a match\", \"column_name\": \"surname\", \"value_l\": \"brown\", \"value_r\": \"doyle\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 3, \"record_number\": 1}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.07217210836446314, \"u_probability\": 0.9813150151128733, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 13.6 times less likely to be a match\", \"column_name\": \"tf_surname\", \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 4, \"record_number\": 1}, {\"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on date of birth\", \"m_probability\": 0.6796640528638345, \"u_probability\": 0.0023542362733926883, \"bayes_factor\": 288.6983182381992, \"log2_bayes_factor\": 8.173418893204259, \"comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `exact match on date of birth` then comparison is 289 times more likely to be a match\", \"column_name\": \"dob\", \"value_l\": \"1862-07-11\", \"value_r\": \"1862-07-11\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 5, \"record_number\": 1}, {\"sql_condition\": \"\\\"postcode_fake_l\\\" = \\\"postcode_fake_r\\\"\", \"label_for_charts\": \"Exact match on full postcode\", \"m_probability\": 0.6753757141159644, \"u_probability\": 0.00014903079407788038, \"bayes_factor\": 4531.786321711654, \"log2_bayes_factor\": 12.14586412290028, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on full postcode` then comparison is 4,532 times more likely to be a match\", \"column_name\": \"postcode_fake\", \"value_l\": \"ba12 0ay\", \"value_r\": \"ba12 0ay\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 6, \"record_number\": 1}, {\"sql_condition\": \"\\\"birth_place_l\\\" = \\\"birth_place_r\\\"\", \"label_for_charts\": \"Exact match on birth_place\", \"m_probability\": 0.8345399212327826, \"u_probability\": 0.005228068621252938, \"bayes_factor\": 159.62681091067626, \"log2_bayes_factor\": 7.318559176669256, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on birth_place` then comparison is 160 times more likely to be a match\", \"column_name\": \"birth_place\", \"value_l\": \"london\", \"value_r\": \"london\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 7, \"record_number\": 1}, {\"sql_condition\": \"\\\"occupation_l\\\" = \\\"occupation_r\\\"\", \"label_for_charts\": \"Exact match on occupation\", \"m_probability\": 0.8982446653774181, \"u_probability\": 0.037284666827814034, \"bayes_factor\": 24.09152989151389, \"log2_bayes_factor\": 4.590454107260654, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on occupation` then comparison is 24.09 times more likely to be a match\", \"column_name\": \"occupation\", \"value_l\": \"politician\", \"value_r\": \"politician\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 8, \"record_number\": 1}, {\"column_name\": \"Final score\", \"label_for_charts\": \"Final score\", \"sql_condition\": null, \"log2_bayes_factor\": 21.13026290618464, \"bayes_factor\": 2295318.468113705, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 9, \"record_number\": 1}, {\"column_name\": \"Prior\", \"label_for_charts\": \"Starting match weight (prior)\", \"sql_condition\": null, \"log2_bayes_factor\": -12.845746707461347, \"bayes_factor\": 0.00013584539607096294, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 0, \"record_number\": 2}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.5457057743059716, \"u_probability\": 0.011951030823329812, \"bayes_factor\": 45.66181632137455, \"log2_bayes_factor\": 5.512916342661154, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 45.66 times more likely to be a match\", \"column_name\": \"first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 1, \"record_number\": 2}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Term freq adjustment on first_name with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on first_name makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 2, \"record_number\": 2}, {\"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.7714400787402513, \"u_probability\": 0.0006788109515158729, \"bayes_factor\": 1136.4579151493144, \"log2_bayes_factor\": 10.150328544506749, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 1,136 times more likely to be a match\", \"column_name\": \"surname\", \"value_l\": \"brown\", \"value_r\": \"brown\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 3, \"record_number\": 2}, {\"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Term freq adjustment on surname with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on surname makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_surname\", \"value_l\": \"brown\", \"value_r\": \"brown\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 4, \"record_number\": 2}, {\"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on date of birth\", \"m_probability\": 0.6796640528638345, \"u_probability\": 0.0023542362733926883, \"bayes_factor\": 288.6983182381992, \"log2_bayes_factor\": 8.173418893204259, \"comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `exact match on date of birth` then comparison is 289 times more likely to be a match\", \"column_name\": \"dob\", \"value_l\": \"1862-07-11\", \"value_r\": \"1862-07-11\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 5, \"record_number\": 2}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.09055579355740942, \"u_probability\": 0.9880262860649464, \"bayes_factor\": 0.0916532230312108, \"log2_bayes_factor\": -3.4476705750282615, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 10.91 times less likely to be a match\", \"column_name\": \"postcode_fake\", \"value_l\": \"ba12 0ay\", \"value_r\": \"tf3 2ng\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 6, \"record_number\": 2}, {\"sql_condition\": \"\\\"birth_place_l\\\" = \\\"birth_place_r\\\"\", \"label_for_charts\": \"Exact match on birth_place\", \"m_probability\": 0.8345399212327826, \"u_probability\": 0.005228068621252938, \"bayes_factor\": 159.62681091067626, \"log2_bayes_factor\": 7.318559176669256, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on birth_place` then comparison is 160 times more likely to be a match\", \"column_name\": \"birth_place\", \"value_l\": \"london\", \"value_r\": \"london\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 7, \"record_number\": 2}, {\"sql_condition\": \"\\\"occupation_l\\\" = \\\"occupation_r\\\"\", \"label_for_charts\": \"Exact match on occupation\", \"m_probability\": 0.8982446653774181, \"u_probability\": 0.037284666827814034, \"bayes_factor\": 24.09152989151389, \"log2_bayes_factor\": 4.590454107260654, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on occupation` then comparison is 24.09 times more likely to be a match\", \"column_name\": \"occupation\", \"value_l\": \"politician\", \"value_r\": \"politician\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 8, \"record_number\": 2}, {\"column_name\": \"Final score\", \"label_for_charts\": \"Final score\", \"sql_condition\": null, \"log2_bayes_factor\": 19.45225978181246, \"bayes_factor\": 717321.2188859307, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 9, \"record_number\": 2}, {\"column_name\": \"Prior\", \"label_for_charts\": \"Starting match weight (prior)\", \"sql_condition\": null, \"log2_bayes_factor\": -12.845746707461347, \"bayes_factor\": 0.00013584539607096294, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 0, \"record_number\": 3}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.5457057743059716, \"u_probability\": 0.011951030823329812, \"bayes_factor\": 45.66181632137455, \"log2_bayes_factor\": 5.512916342661154, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 45.66 times more likely to be a match\", \"column_name\": \"first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 1, \"record_number\": 3}, {\"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Term freq adjustment on first_name with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on first_name makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_first_name\", \"value_l\": \"sarah\", \"value_r\": \"sarah\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 2, \"record_number\": 3}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.07217210836446314, \"u_probability\": 0.9813150151128733, \"bayes_factor\": 0.07354632024677796, \"log2_bayes_factor\": -3.765203029049615, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 13.6 times less likely to be a match\", \"column_name\": \"surname\", \"value_l\": \"brown\", \"value_r\": \"doyle\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 3, \"record_number\": 3}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.07217210836446314, \"u_probability\": 0.9813150151128733, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 13.6 times less likely to be a match\", \"column_name\": \"tf_surname\", \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 4, \"record_number\": 3}, {\"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on date of birth\", \"m_probability\": 0.6796640528638345, \"u_probability\": 0.0023542362733926883, \"bayes_factor\": 288.6983182381992, \"log2_bayes_factor\": 8.173418893204259, \"comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `exact match on date of birth` then comparison is 289 times more likely to be a match\", \"column_name\": \"dob\", \"value_l\": \"1862-07-11\", \"value_r\": \"1862-07-11\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 5, \"record_number\": 3}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.09055579355740942, \"u_probability\": 0.9880262860649464, \"bayes_factor\": 0.0916532230312108, \"log2_bayes_factor\": -3.4476705750282615, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 10.91 times less likely to be a match\", \"column_name\": \"postcode_fake\", \"value_l\": \"ba12 0ay\", \"value_r\": \"tf3 2ng\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 6, \"record_number\": 3}, {\"sql_condition\": \"\\\"birth_place_l\\\" = \\\"birth_place_r\\\"\", \"label_for_charts\": \"Exact match on birth_place\", \"m_probability\": 0.8345399212327826, \"u_probability\": 0.005228068621252938, \"bayes_factor\": 159.62681091067626, \"log2_bayes_factor\": 7.318559176669256, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on birth_place` then comparison is 160 times more likely to be a match\", \"column_name\": \"birth_place\", \"value_l\": \"london\", \"value_r\": \"london\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 7, \"record_number\": 3}, {\"sql_condition\": \"\\\"occupation_l\\\" = \\\"occupation_r\\\"\", \"label_for_charts\": \"Exact match on occupation\", \"m_probability\": 0.8982446653774181, \"u_probability\": 0.037284666827814034, \"bayes_factor\": 24.09152989151389, \"log2_bayes_factor\": 4.590454107260654, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on occupation` then comparison is 24.09 times more likely to be a match\", \"column_name\": \"occupation\", \"value_l\": \"politician\", \"value_r\": \"politician\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 8, \"record_number\": 3}, {\"column_name\": \"Final score\", \"label_for_charts\": \"Final score\", \"sql_condition\": null, \"log2_bayes_factor\": 5.5367282082561, \"bayes_factor\": 46.421724360169, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 9, \"record_number\": 3}, {\"column_name\": \"Prior\", \"label_for_charts\": \"Starting match weight (prior)\", \"sql_condition\": null, \"log2_bayes_factor\": -12.845746707461347, \"bayes_factor\": 0.00013584539607096294, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 0, \"record_number\": 4}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.19797580211251312, \"u_probability\": 0.9619834090296856, \"bayes_factor\": 0.20579960138003153, \"log2_bayes_factor\": -2.2806879070559116, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 4.859 times less likely to be a match\", \"column_name\": \"first_name\", \"value_l\": \"sarah\", \"value_r\": \"jane\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 1, \"record_number\": 4}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.19797580211251312, \"u_probability\": 0.9619834090296856, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 4.859 times less likely to be a match\", \"column_name\": \"tf_first_name\", \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 2, \"record_number\": 4}, {\"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.7714400787402513, \"u_probability\": 0.0006788109515158729, \"bayes_factor\": 1136.4579151493144, \"log2_bayes_factor\": 10.150328544506749, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 1,136 times more likely to be a match\", \"column_name\": \"surname\", \"value_l\": \"brown\", \"value_r\": \"brown\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 3, \"record_number\": 4}, {\"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Term freq adjustment on surname with weight {cl.tf_adjustment_weight}\", \"m_probability\": null, \"u_probability\": null, \"bayes_factor\": 1.0, \"log2_bayes_factor\": 0.0, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"Term frequency adjustment on surname makes comparison 1.00 times more likely to be a match\", \"column_name\": \"tf_surname\", \"value_l\": \"brown\", \"value_r\": \"brown\", \"term_frequency_adjustment\": true, \"bar_sort_order\": 4, \"record_number\": 4}, {\"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 315576000.0\", \"label_for_charts\": \"Abs date difference <= 10 year\", \"m_probability\": 0.02726057331431288, \"u_probability\": 0.25042855211572257, \"bayes_factor\": 0.10885569190894744, \"log2_bayes_factor\": -3.1995112489219304, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `abs date difference <= 10 year` then comparison is 9.186 times less likely to be a match\", \"column_name\": \"dob\", \"value_l\": \"1862-07-11\", \"value_r\": \"1860-01-01\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 5, \"record_number\": 4}, {\"sql_condition\": \"\\\"postcode_fake_l\\\" = \\\"postcode_fake_r\\\"\", \"label_for_charts\": \"Exact match on full postcode\", \"m_probability\": 0.6753757141159644, \"u_probability\": 0.00014903079407788038, \"bayes_factor\": 4531.786321711654, \"log2_bayes_factor\": 12.14586412290028, \"comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on full postcode` then comparison is 4,532 times more likely to be a match\", \"column_name\": \"postcode_fake\", \"value_l\": \"ba12 0ay\", \"value_r\": \"ba12 0ay\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 6, \"record_number\": 4}, {\"sql_condition\": \"\\\"birth_place_l\\\" = \\\"birth_place_r\\\"\", \"label_for_charts\": \"Exact match on birth_place\", \"m_probability\": 0.8345399212327826, \"u_probability\": 0.005228068621252938, \"bayes_factor\": 159.62681091067626, \"log2_bayes_factor\": 7.318559176669256, \"comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on birth_place` then comparison is 160 times more likely to be a match\", \"column_name\": \"birth_place\", \"value_l\": \"london\", \"value_r\": \"london\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 7, \"record_number\": 4}, {\"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.10175533462258204, \"u_probability\": 0.9627153331721859, \"bayes_factor\": 0.10569618153612875, \"log2_bayes_factor\": -3.2420048371677, \"comparison_vector_value\": 0, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is 9.461 times less likely to be a match\", \"column_name\": \"occupation\", \"value_l\": \"politician\", \"value_r\": \"artist\", \"term_frequency_adjustment\": false, \"bar_sort_order\": 8, \"record_number\": 4}, {\"column_name\": \"Final score\", \"label_for_charts\": \"Final score\", \"sql_condition\": null, \"log2_bayes_factor\": 8.046801143469395, \"bayes_factor\": 264.44083104551703, \"comparison_vector_value\": null, \"m_probability\": null, \"u_probability\": null, \"bayes_factor_description\": null, \"value_l\": \"\", \"value_r\": \"\", \"term_frequency_adjustment\": null, \"bar_sort_order\": 9, \"record_number\": 4}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.LayerChart(...)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "records_to_plot = comparisons.to_dict(orient=\"records\")\n",
    "linker.visualisations.waterfall_chart(records_to_plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a closer look at each comparison helps reveal which features have the most impact in terms of the hypothesis. Some key takeaways are:\n",
    "\n",
    "- Matching on postcode and surname are the strongest indicators of a link. Therefore, when both surname and postcode change, you lose that strong link advantage. It's not that a non-match on surname or postcode is overly detrimental to the records being linked, but without the boost from a surname or postcode match, they only meet the low linkage threshold. \n",
    "\n",
    "- In the sibling scenario, you can see the impact of one or two strong predictors in a model can have. Even with three non-matches (first name, DOB, and occupation), matching on both postcode and surname links the records at a medium threshold. \n",
    "\n",
    "- As first name matches are less predictive than surname matches the sibling scenario has a more predictive name match than the same person changing their information. \n",
    "\n",
    "- Due to the way individual factors are weighted, the combination of non-matches in this scenario means that people who change both their postcode and surname won't be linked at higher thresholds. Given this affects women more than men in our data, it leads to a **gender bias** in the pipeline.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **4. Identify mitigations**\n",
    "\n",
    "Now that bias has been detected, it's time to explore potential mitigation strategies:\n",
    "\n",
    "<u>Are there viable technical solutions?</u>\n",
    "\n",
    "In our example, a change in surname and postcode allows records to be linked at a low threshold. We should consider what can be done in the pipeline to ensure these comparisons are linked.\n",
    "\n",
    "Any technical solution should align with the hypothesis, as opposed to just pushing comparisons randomly over/under a threshold. Different elements of the pipeline can be altered to address the bias. \n",
    "\n",
    "- Altering the _input data_ - there's likely no potential solution here, as the bias stems from legitimate data qualities, not errors. \n",
    "- Updating the _model design_ - a TF adjustment might alter results, but this would be random so wouldn't address the hypothesis. However, **adjusting comparison levels** might help. \n",
    "- Adjusting the _output data_ - since the records link at a low threshold, **lowering the threshold** across the model would result in a link.\n",
    "\n",
    "<u>Could they negatively impact overall performance?</u>\n",
    "\n",
    "**Lowering the threshold**\n",
    "\n",
    "This could link records with surname and postcode changes but might also increase false positives, impacting overall performance. This trade-off needs careful consideration. In synthetic data, for instance, siblings would link at low thresholds. This is just one scenario, and there are most likely others. Given these potential issues, we’re not pursuing a lower threshold.\n",
    "\n",
    "**Adjusting comparison levels**\n",
    "\n",
    "We'd need to decide that a factor’s predictive power doesn't fit our hypothesis - but also make sure this applies across the whole model. There are lots of options, like making surname or postcode less predictive, or boosting factors like occupation, first name, DOB, or birthplace. We'll focus on one example to demonstrate the necessary considerations. \n",
    "\n",
    "Say we decide to manually make first name as predictive as surname, and we find that this change won’t impact overall performance. Therefore, we can take this technical solution to the next consideration.\n",
    "\n",
    "<u>Could they introduce further bias?</u>\n",
    "\n",
    "It's important to think about other groups that this decision could impact. \n",
    "\n",
    "In this dataset, there's records relating to a community of Vietnamese people. They share similar postcodes and often list the same birthplace (because the dataset defaults to the country of birth if they were born outside the UK). Many families in this community have the same surname, as do many unrelated individuals (due to less variation in Vietnamese surnames). Additionally, because Vietnamese naming conventions differ from Western ones, surnames are often incorrectly recorded as first names in this dataset. \n",
    "\n",
    "Therefore, increasing the predictive power of first name matches will likely lead to more false positives for this community, introducing bias against the Vietnamese population in the dataset. The decision is made to not attempt bias mitigation.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **5. Make a statement about bias** \n",
    "\n",
    "When a bias is detected but not mitigated, it’s crucial to explore it further to gain more insight. This helps refine the understanding of the bias and lays the groundwork for impact assessment.\n",
    "\n",
    "In our example, we can dig deeper into how the link fails with postcode and surname changes. To do this, we’ll create some more synthetic data with partial changes in surname and postcode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "synthetic_comparison_partial_raw = [\n",
    "    {\"unique_id\": 2, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"brown-doyle\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"tf3 2ng\", \"gender\": \"female\", \"occupation\": \"politician\"},\n",
    "    {\"unique_id\": 3, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"doyle\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"ba13 2ng\", \"gender\": \"female\", \"occupation\": \"politician\"},\n",
    "    {\"unique_id\": 4, \"person_id\": 1, \"first_name\": \"sarah\", \"surname\": \"brown-doyle\", \"dob\": \"1862-07-11\", \"birth_place\": \"london\", \"postcode_fake\": \"ba13 2ng\", \"gender\": \"female\", \"occupation\": \"politician\"}\n",
    "]\n",
    "\n",
    "synthetic_comparison_partial_df = pd.DataFrame(synthetic_comparison_partial_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this scenario, we generate new records to be compared to the base. These represent a partial surname change (double-barrel) and a full postcode change, a partial postcode change (same area, different district) with a full surname change, and a combination of partial postcode and partial surname changes.\n",
    "\n",
    "We can then use the same model to link these records and examine the resulting match probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"first_name\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n",
      "No term frequencies found for column \"surname\".\n",
      "To apply term frequency adjustments, you need to register a lookup using `linker.table_management.register_term_frequency_lookup`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_b9bde_row0_col1, #T_b9bde_row1_col1, #T_b9bde_row2_col1 {\n",
       "  background-color: #c4f5bf;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_b9bde\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_b9bde_level0_col0\" class=\"col_heading level0 col0\" >match_weight</th>\n",
       "      <th id=\"T_b9bde_level0_col1\" class=\"col_heading level0 col1\" >match_probability</th>\n",
       "      <th id=\"T_b9bde_level0_col2\" class=\"col_heading level0 col2\" >unique_id_l</th>\n",
       "      <th id=\"T_b9bde_level0_col3\" class=\"col_heading level0 col3\" >unique_id_r</th>\n",
       "      <th id=\"T_b9bde_level0_col4\" class=\"col_heading level0 col4\" >first_name_l</th>\n",
       "      <th id=\"T_b9bde_level0_col5\" class=\"col_heading level0 col5\" >first_name_r</th>\n",
       "      <th id=\"T_b9bde_level0_col6\" class=\"col_heading level0 col6\" >surname_l</th>\n",
       "      <th id=\"T_b9bde_level0_col7\" class=\"col_heading level0 col7\" >surname_r</th>\n",
       "      <th id=\"T_b9bde_level0_col8\" class=\"col_heading level0 col8\" >dob_l</th>\n",
       "      <th id=\"T_b9bde_level0_col9\" class=\"col_heading level0 col9\" >dob_r</th>\n",
       "      <th id=\"T_b9bde_level0_col10\" class=\"col_heading level0 col10\" >postcode_fake_l</th>\n",
       "      <th id=\"T_b9bde_level0_col11\" class=\"col_heading level0 col11\" >postcode_fake_r</th>\n",
       "      <th id=\"T_b9bde_level0_col12\" class=\"col_heading level0 col12\" >birth_place_l</th>\n",
       "      <th id=\"T_b9bde_level0_col13\" class=\"col_heading level0 col13\" >birth_place_r</th>\n",
       "      <th id=\"T_b9bde_level0_col14\" class=\"col_heading level0 col14\" >occupation_l</th>\n",
       "      <th id=\"T_b9bde_level0_col15\" class=\"col_heading level0 col15\" >occupation_r</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_b9bde_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_b9bde_row0_col0\" class=\"data row0 col0\" >15.641700</td>\n",
       "      <td id=\"T_b9bde_row0_col1\" class=\"data row0 col1\" >0.999980</td>\n",
       "      <td id=\"T_b9bde_row0_col2\" class=\"data row0 col2\" >1</td>\n",
       "      <td id=\"T_b9bde_row0_col3\" class=\"data row0 col3\" >2</td>\n",
       "      <td id=\"T_b9bde_row0_col4\" class=\"data row0 col4\" >sarah</td>\n",
       "      <td id=\"T_b9bde_row0_col5\" class=\"data row0 col5\" >sarah</td>\n",
       "      <td id=\"T_b9bde_row0_col6\" class=\"data row0 col6\" >brown</td>\n",
       "      <td id=\"T_b9bde_row0_col7\" class=\"data row0 col7\" >brown-doyle</td>\n",
       "      <td id=\"T_b9bde_row0_col8\" class=\"data row0 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_b9bde_row0_col9\" class=\"data row0 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_b9bde_row0_col10\" class=\"data row0 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_b9bde_row0_col11\" class=\"data row0 col11\" >tf3 2ng</td>\n",
       "      <td id=\"T_b9bde_row0_col12\" class=\"data row0 col12\" >london</td>\n",
       "      <td id=\"T_b9bde_row0_col13\" class=\"data row0 col13\" >london</td>\n",
       "      <td id=\"T_b9bde_row0_col14\" class=\"data row0 col14\" >politician</td>\n",
       "      <td id=\"T_b9bde_row0_col15\" class=\"data row0 col15\" >politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b9bde_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_b9bde_row1_col0\" class=\"data row1 col0\" >12.130717</td>\n",
       "      <td id=\"T_b9bde_row1_col1\" class=\"data row1 col1\" >0.999777</td>\n",
       "      <td id=\"T_b9bde_row1_col2\" class=\"data row1 col2\" >1</td>\n",
       "      <td id=\"T_b9bde_row1_col3\" class=\"data row1 col3\" >3</td>\n",
       "      <td id=\"T_b9bde_row1_col4\" class=\"data row1 col4\" >sarah</td>\n",
       "      <td id=\"T_b9bde_row1_col5\" class=\"data row1 col5\" >sarah</td>\n",
       "      <td id=\"T_b9bde_row1_col6\" class=\"data row1 col6\" >brown</td>\n",
       "      <td id=\"T_b9bde_row1_col7\" class=\"data row1 col7\" >doyle</td>\n",
       "      <td id=\"T_b9bde_row1_col8\" class=\"data row1 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_b9bde_row1_col9\" class=\"data row1 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_b9bde_row1_col10\" class=\"data row1 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_b9bde_row1_col11\" class=\"data row1 col11\" >ba13 2ng</td>\n",
       "      <td id=\"T_b9bde_row1_col12\" class=\"data row1 col12\" >london</td>\n",
       "      <td id=\"T_b9bde_row1_col13\" class=\"data row1 col13\" >london</td>\n",
       "      <td id=\"T_b9bde_row1_col14\" class=\"data row1 col14\" >politician</td>\n",
       "      <td id=\"T_b9bde_row1_col15\" class=\"data row1 col15\" >politician</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b9bde_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_b9bde_row2_col0\" class=\"data row2 col0\" >22.235689</td>\n",
       "      <td id=\"T_b9bde_row2_col1\" class=\"data row2 col1\" >1.000000</td>\n",
       "      <td id=\"T_b9bde_row2_col2\" class=\"data row2 col2\" >1</td>\n",
       "      <td id=\"T_b9bde_row2_col3\" class=\"data row2 col3\" >4</td>\n",
       "      <td id=\"T_b9bde_row2_col4\" class=\"data row2 col4\" >sarah</td>\n",
       "      <td id=\"T_b9bde_row2_col5\" class=\"data row2 col5\" >sarah</td>\n",
       "      <td id=\"T_b9bde_row2_col6\" class=\"data row2 col6\" >brown</td>\n",
       "      <td id=\"T_b9bde_row2_col7\" class=\"data row2 col7\" >brown-doyle</td>\n",
       "      <td id=\"T_b9bde_row2_col8\" class=\"data row2 col8\" >1862-07-11</td>\n",
       "      <td id=\"T_b9bde_row2_col9\" class=\"data row2 col9\" >1862-07-11</td>\n",
       "      <td id=\"T_b9bde_row2_col10\" class=\"data row2 col10\" >ba12 0ay</td>\n",
       "      <td id=\"T_b9bde_row2_col11\" class=\"data row2 col11\" >ba13 2ng</td>\n",
       "      <td id=\"T_b9bde_row2_col12\" class=\"data row2 col12\" >london</td>\n",
       "      <td id=\"T_b9bde_row2_col13\" class=\"data row2 col13\" >london</td>\n",
       "      <td id=\"T_b9bde_row2_col14\" class=\"data row2 col14\" >politician</td>\n",
       "      <td id=\"T_b9bde_row2_col15\" class=\"data row2 col15\" >politician</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f885a805d00>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comparisons_partial = compare_records(synthetic_base_raw, synthetic_comparison_partial_raw, linker)\n",
    "\n",
    "comparisons_partial[columns_of_interest].style.map(highlight_cells, subset=['match_probability'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In all three cases, the records will remain linked at a high threshold. This insight can help make the final bias statement:\n",
    "\n",
    "_Given how the model weighs the different parameters to produce match probabilities, a full change in both surname and postcode will lead to a non-link. As women are more likely than men to make these changes in the input data, this will introduce a gender bias in the pipeline._\n",
    "\n",
    "Now that this bias has been detected, it's important to note that we can't yet determine its exact impact on the linked data. This is because the final results are influenced by many other factors in the pipeline, which were intentionally excluded from this process to isolate the bias. \n",
    "\n",
    "Further investigation is needed to understand impact — such as understanding how many records in the input data undergo these changes, or how many records with a full surname and postcode change end up linked or not linked in the final data. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
